Autonomous vehicle (AV) navigation in some cases uses lidar and cameras to create a high resolution three-dimensional (3D) model of the surroundings of the vehicle. The 3D model is formed by combining information from multiple sensors in a system to generate a 3D point cloud. The point cloud, along with other sensor data including cameras, may then be fed into a perception system which will detect and classify objects relevant to the driving task. A similar approach can be used outside of AV navigation for a variety of mobile and fixed observation systems.
Lidar can generate a 3D position for any object that reflects the illumination from the lidar. These positions can then be used as the basis of a 3D point cloud. The cloud can be used as the 3D model or combined with other sensor data and analysis to generate the 3D model of the surroundings.
Visible and near infrared light cameras are small inexpensive sensors that provide detailed information at a high sampling rate. The detail is particularly helpful in object detection, identification, and localization. The high sampling rate is particularly helpful in determining motion (motion vector generation). However, cameras do not provide accurate or reliable range information and they do not image objects that are not emitting or reflecting light. When camera data is added to lidar data a much more complete understanding of the scene can be obtained. Combining lidar and camera data is sometimes referred to as fusion.